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EXECUTIVE  SUMMARY 


This  report  describes  in  detail  the  procedures,  methods,  and  resources  used  to  complete  the 
demonstration  project  at  the  fonner  Pole  Mountain  Target  and  Maneuver  Area  (PMTMA) 
(herein  referred  to  as  Pole  Mountain)  for  Environmental  Security  Technology  Certification 
Program  (ESTCP)  Project  MR-201167  (USACE  Participation  in  the  Pole  Mountain  Advanced 
Classification  Demonstration).  The  objective  of  U.S.  Army  Corps  of  Engineers  (USACE) 
participation  was  to  learn  to  use  advanced  classification  processes  and  apply  them  to  making 
dig/no-dig  decisions  on  munitions  response  sites. 

USACE’s  responsibilities  on  this  project  included  only  the  processing  and  analysis  of 
MetalMapper  data  collected  at  the  Pole  Mountain  site  by  Sky  Research,  Inc  (Sky).  The 
MetalMapper  is  an  advanced  electromagnetic  induction  system  developed  by  Geometries,  Inc 
(Geometries),  with  support  from  the  ESTCP.  It  has  three  mutually  orthogonal  transmit  loops  in 
the  Z,  Y,  and  X  directions  and  contains  seven  tri-axial  receiver  antennas  inside  the  Z  (bottom) 
loop,  allowing  a  total  of  63  independent  measurements  of  the  transient  secondary  magnetic  field 
when  all  transmitter  coils  are  used.  Sky  personnel  collected  MetalMapper  data  over  2,370  targets 
in  a  static  mode,  preprocessed  the  data,  and  submitted  background  corrected  .CSV  files  for  each 
target  to  ESTCP,  who  then  forwarded  the  files  to  the  USACE. 

The  2,370  data  files  were  inverted  and  analyzed  using  the  UX-Analyze  add-on  to  Geosoft’s 
Oasis  Montaj  software  package.  Once  analysis  was  complete,  theoretical  ranked  dig  lists 
(theoretical  because  all  targets  were  intrusively  investigated  regardless  of  the  indicated  stop  dig 
points)  were  submitted  for  scoring  by  the  Institute  for  Defense  Analyses  (IDA).  Dig  list  scoring 
was  based  on  the  number  of  targets  of  interest  (TOIs)  correctly  identified  as  items  that  should  be 
dug  and  the  number  of  non-TOI  or  clutter  items  that  were  correctly  classified  as  items  that  did 
not  need  to  be  intrusively  investigated.  Dig  lists  submitted  by  USACE  were  scored  against  the 
ground  truth  data  generated  during  the  intrusive  investigation  perfonned  at  the  site  following 
MetalMapper  data  collection. 

The  dig  list  submitted  by  USACE  Omaha  District  correctly  identified  all  of  the  TOI  on  site  as 
TOI  and  the  number  of  false  positives  was  reduced  by  approximately  59%.  A  retrospective 
analysis  was  performed  to  examine  ways  of  improving  the  results  of  the  advanced  classification. 
Results  indicate  that  more  aggressive  use  of  a  size  parameter  leads  to  a  reduction  of  false 
positives  by  up  to  77%. 
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1.0  INTRODUCTION 


Remedial  actions  at  Munitions  Response  Sites  often  involve  the  systematic  surveying  of  the  site 
with  a  geophysical  sensor  integrated  with  a  global  positioning  system  (GPS).  This  data  is  used  to 
construct  maps  from  which  anomalous  responses,  otherwise  known  as  anomalies,  are  identified. 
The  locations  of  all  anomalies  are  then  dug.  A  substantial  amount  of  time  and  money  is  spent 
excavating  non-hazardous  pieces  of  metal  on  munitions  response  sites.  In  some  cases  over  95% 
of  the  remedial  action  funds  are  expended  digging  fragments,  clutter  and  cultural  debris  that 
could  be  left  in  the  ground  without  adding  inherent  risk  to  the  land  users.  Leaving  non- 
hazardous  material  in  the  ground  provides  the  potential  for  a  significant  cost  saving. 

Advanced  classification  as  applied  to  munitions  response  refers  to  a  process  used  to  make  a 
decision  regarding  whether  the  source  of  the  anomalies  are  hazardous  or  not.  Advanced 
classification  is  based  on  target  parameters  derived  from  fitting  physics-based  models  to  the 
observed  geophysical  sensor  responses.  Advanced  classification  takes  advantage  of  more  recent 
advanced  electromagnetic  induction  (EMI)  sensors,  specifically  the  MetalMapper,  TEMTADS 
and  Berkley  Unexploded  Ordnance  (UXO)  Discriminator,  which  are  capable  of  collecting  static, 
high  resolution,  three-dimensional  data  over  individual  targets.  This  data  is  then  inverted  or 
modeled  to  produce  polarizability  curves  that  are  inherent  to  the  object.  By  clustering  like 
curves  and/or  matching  these  curves  to  a  library  of  know  objects  the  processer  can  classify  each 
target  as  a  target  of  interest  (TOI)  or  not. 

The  Environmental  Security  Technology  Certification  Program  (ESTCP)  initiated  a 
Classification  Pilot  Program  in  2007  to  validate  the  application  of  a  number  of  recently 
developed  technologies  in  a  comprehensive  approach  to  munitions  response.  The  goal  of  the 
program  is  to  demonstrate  that  classification  decisions  can  be  made  using  an  explicit  approach, 
based  on  principled  physics-based  analysis  that  is  transparent  and  reproducible.  The  former  Pole 
Mountain  Target  and  Maneuver  Area  (herein  referred  to  as  Pole  Mountain)  site  is  the  sixth  in  a 
series  of  demonstration  sites  designed  to  showcase  the  use  of  advanced  classification 
technologies  in  support  of  the  Military  Munitions  Response  Program. 

1.1  OBJECTIVE  OF  THE  DEMONSTRATION 

The  objective  of  the  U.S.  Anny  Corps  of  Engineers  (USACE)  participation  in  the  advanced 
classification  demonstration  project  was  to  learn  to  use  advanced  classification  processes  and 
apply  them  to  making  dig/no-dig  decisions  on  munitions  response  sites.  Initially,  EMI  data  was 
collected  by  URS  Corporation  (URS)  using  a  single-sensor  Geonics,  Inc  (Geonics)  EM61-MK2 
in  cart  configuration  as  shown  in  Figure  1. 
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Figure  1:  EM61-MK2  Cart  Used  to  Collect  Data  at  the  Pole  Mountain  Site 


The  EM61-MK2  data  was  preprocessed  by  URS  and  a  total  of  2,370  anomalies  were  identified. 
A  Geometries,  Inc  (Geometries)  MetalMapper  sensor  was  used  by  Sky  Research,  Inc  (Sky)  to 
perfonn  cued  interrogation  on  all  the  identified  anomalies.  The  MetalMapper  sensor  was 
mounted  to  a  Kubota  All-Terrain  Vehicle  as  shown  in  Figure  2. 


Figure  2:  MetalMapper  Setup  Used  to  Collect  Data  at  the  Pole  Mountain  Site. 


The  MetalMapper  data  was  preprocessed  by  Sky  through  the  stage  of  background  correction,  and 
then  made  available  to  the  participants  of  this  demonstration  along  with  the  EM61-MK2  data  and 
the  list  of  anomalies. 

The  MetalMapper  data  was  processed  by  USACE  using  Oasis  Montaj  UX-Analyze  algorithms. 
Dig/no-dig  decisions  were  made  by  categorizing  the  target  list  based  on  specific  parameters 
which  are  discussed  below. 
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2.0  TECHNOLOGY 


2.1  METALMAPPER  TECHNOLOGY  DESCRIPTION 

The  MetalMapper  is  an  advanced  EMI  system  developed  by  Geometries,  with  support  from  the 
ESTCP.  The  MetalMapper  system  uses  time  domain  electromagnetic  (TDEM)  principles  to 
induce  electrical  currents  in  buried  metallic  objects  and  then  measure  the  effects  of  those  currents 
in  receivers  located  on  the  ground  surface.  It  has  3  orthogonal  transmitter  coils,  each 
approximately  1  m  x  1  m  in  size.  One  coil  is  oriented  horizontally  to  generate  vertical  fields  and 
the  two  other  coils  are  mounted  vertically  orthogonal  to  each  other  as  shown  in  Figure  3.  Within 
the  box  containing  the  horizontal  coil  are  7  receiver  cubes,  each  one  containing  3  orthogonal 
coils  to  measure  the  fields,  thus  resulting  in  21  different  receiver  coils.  The  receiver  coils  are 
oriented  in  the  same  manner  as  the  transmitter  coils. 


Figure  3:  Assembled  MetalMapper  Unit. 


The  transmitter  coils  are  powered  using  a  bi-polar  half  duty  cycle  and  the  time  decay  of  the 
subsurface  currents  (transients)  are  measured  during  the  off  time  of  the  transmitter  coils.  The 
transmitter  coils  are  activated  in  sequence  and  measurements  are  recorded  in  all  21  receiver 
coils.  In  the  case  of  Pole  Mountain  where  all  transmitter  coils  were  used,  this  resulted  in  63 
different  EM  transients  measured  and  recorded. 

The  MetalMapper  has  two  modes  of  data  collection,  dynamic  and  static.  Dynamic  mode  data  are 
collected  while  the  antenna  platform  is  in  motion.  Static  mode  data  collection  is  employed  for 
cued  surveys.  As  the  name  implies,  the  antenna  platform  remains  static  or  motionless  during  the 
period  of  data  acquisition.  Depending  on  the  acquisition  parameters  (e.g.  sample  period  and 
stacking  parameter)  it  can  take  tens  of  seconds  to  complete  a  single  static  measurement.  The 
results  of  the  static  measurement  are  written  into  a  binary  data  file  containing  only  a  single  data 
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point  representing  the  average  (stacked)  result,  usually  over  tens  or  even  hundreds  of  repetitions 
of  the  transmitter’s  base  frequency. 

Data  are  acquired  in  time  blocks  that  consist  of  a  fixed  number  of  transmitter  cycle  “repeats”. 
Both  the  period  and  the  repeat  factor  are  operator  selectable  and  are  varied  in  multiplicative 
factors  of  3.  The  MetalMapper  also  averages  an  operator-specified  number  of  acquisition  blocks 
(NStacks)  together  before  the  acquired  data  are  saved  to  disk.  The  decay  transients  received 
during  the  off  times  are  stacked  (averaged)  with  appropriate  sign  changes  for  positive  and 
negative  half  cycles.  The  decays  in  an  individual  acquisition  block  are  stacked,  and  the  decays  in 
that  block  are  averaged  with  other  acquisition  blocks  (assuming  the  operator  has  selected  NStack 
greater  than  1).  The  resultant  data  are  saved  as  a  data  point. 

In  its  present  (third  generation)  fonn,  the  MetalMapper  technology  has  been  demonstrated  and 
scored  at  the  Standardized  UXO  Technology  Demonstration  sites  at  Yuma  Proving  Grounds 
(Blind  Grid  only),  at  Aberdeen  Proving  Grounds  (Blind  Grid  plus  Direct  Fire  and  Indirect  Fire 
Areas),  and  most  recently  at  Camp  San  Luis  Obispo  (SLO),  Camp  Butner,  and  Camp  Beale  in 
connection  with  2009  through  2011  Classification  Studies  carried  out  by  ESTCP.  The 
perfonnance  of  the  MetalMapper  at  these  sites  is  documented  in  fonnal  reports  issued  by  the 
Aberdeen  Test  Center  and  by  the  various  demonstrators  who  analyzed  the  data  collected  at  Camp 
SLO,  Camp  Butner,  and  Camp  Beale. 

2.2  UX-ANALYZE  SOFTWARE  TECHNOLOGY  DESCRIPTION 

The  UX- Analyze  module  in  Geosoft  Oasis  montaj  software  was  the  primary  tool  used  to  analyze 
MetalMapper  data  during  USACE’s  participation  in  this  advanced  classification  demonstration. 
UX-Analyze  is  a  target  selection,  fitting  and  classification  tool  for  UXO  applications  (Figure  4). 
During  the  course  of  several  SERDP  and  ESTCP  projects,  AETC  and  Duke  University 
developed  various  advanced  processing  procedures  for  improved  detection  of  buried  UXO  and 
discrimination  between  UXO  and  clutter.  Different  procedures  have  been  developed  for  use  with 
magnetometer  data  and  EMI  sensor  data.  The  procedures  rely  on  physics-based  models  for  the 
sensor  response  due  to  buried  objects  and  estimate  model  parameters  that  correlate  with  target 
features  or  location  to  produce  an  optimal  match  between  the  modeled-  and  measured-sensor 
responses.  Target  location,  depth,  and  magnetic  dipole  moment  can  be  determined  from 
magnetometer  survey  data,  and  the  size  of  the  target  can  be  estimated  from  the  dipole  moment. 
These  parameters  have  proven  to  be  useful  for  discriminating  between  buried  UXO  and  some 
clutter  items.  With  proper  processing,  EMI  sensor  data  collected  above  an  unknown  object  can 
be  used  to  detennine  eigenvalues  of  the  magnetic  polarizability  tensor,  which  in  turn  can  be  used 
to  determine  infonnation  regarding  the  object's  shape,  size  and  burial  depth.  Robust,  statistically 
efficient  decision  rules  for  target  classification  and  discrimination  then  can  be  constructed  using 
any  of  the  target  features. 
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Figure  4:  Screen  Snapshot  of  Computer  Monitor  during  Data  Analysis  using  UX-Analyze. 
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3.0  PERFORMANCE  OBJECTIVES 


The  specific  performance  objectives  for  this  demonstration  are  based  on  the  objectives  stated  in 
the  Project  Demonstration  Plan  (U.S.  Anny  Corps  of  Engineers,  2011)  and  are  summarized  in 

Table  1. 


Table  1:  Performance  Objectives  for  this  Demonstration. 


Performance  Objective 

Metric 

Data  Required 

Success  Criteria 

Correct  classification  of 
munitions 

Number  of  UXO  correctly 
identified 

•  Anomaly 
Classification  List 

•  Ground  truth 

Approach  correctly  identifies 
all  UXO  down  to  30  cm  as 

TOI  and  97%  of  all  UXO  57- 
mm  or  larger  when  deeper 
than  30  cm. 

Maximize  correct 
classification  of  non¬ 
munitions 

Number  of  non-UXO 
correctly  identified 

•  Anomaly 
Classification  List 

•  Ground  truth 

Approach  correctly  identifies 
75%  non-UXO  anomalies  as 
not  TOI. 

Stop  dig  threshold 

Number  of  UXO  recovered 
and  number  of  false  alarms 
eliminated 

•  Prioritized  Dig  List 

•  Prioritized  No-  Dig 
List 

•  Scoring  reports 
from  the  IDA 

Threshold  between  dig  and 
no-dig  places  50%  of  the 
master  list  of  anomalies  in  the 
no-dig  category  while 
identifying  all  UXO  30  cm 
deep  or  shallower  as  TOI  and 
97%  or  more  of  UXO  57-mm 
or  larger  as  TOI  when  deeper 
than  30  cm.  (Assumes  ~10% 
of  all  anomalies  on  master  list 
are  UXO.) 

Minimize  number  of 
anomalies  that  cannot  be 
analyzed 

Number  of  anomalies  that 
must  be  classified  as  “Unable 
to  apply  decision  rules”. 

•  Demonstrator 

decision  rules  and 
parameters. 

Reliable  target  parameters 
can  be  estimated  and  decision 
rales  applied  to  90%  or  more 
of  anomalies  on  master 
anomaly  list. 

Correct  estimation  of  target 
parameters 

Accuracy  of  estimated  target 
parameters. 

•  Calculated  target 
parameters 

•  Results  of  intrusive 
investigation 

X,  Y<  15  cm  (16),  Z  <  10  cm 
(16),  size  ±  20%,  symmetry 
estimate  correct  >  95%  for 
anomalies  processed 

3.1  CORRECT  CLASSIFICATION  OF  MUNITIONS 

This  is  one  of  the  two  primary  measures  of  the  effectiveness.  By  collecting  high-quality  data  and 
analyzing  those  data  with  advanced  parameter  estimation,  we  anticipate  classification  algorithms 
and  rule-based  decisions  will  correctly  classify  UXO  targets  in  the  TOI  class. 

3.1.1  Metric 

The  metric  for  this  objective  is  the  number  of  items  on  the  master  anomaly  list  that  are  correctly 
classified  as  UXO  and  placed  on  the  dig  list. 

3.1.2  Data  Requirements 

A  list  of  anomalies  from  the  master  anomaly  list  classified  as  TOI  and  ground  truth  for  those 
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anomalies. 

3.1.3  Success  Criteria 

The  objective  will  be  considered  to  be  met  if  100%  of  all  UXO  30  cm  deep  (to  center  of  mass)  or 
shallower  are  correctly  identified  as  TOI  and  if  97%  or  more  of  the  UXO  57-mm  in  diameter  or 
larger  are  correctly  identified  as  TOI  when  deeper  than  30  cm.  Items  smaller  than  57-mm  in 
diameter  and  deeper  than  30  cm  (to  center  of  mass)  are  not  anticipated.  If  these  items  are  present 
but  not  selected  in  the  TOI  class,  they  will  not  be  counted  in  this  metric  calculation. 

3.2  MAXIMIZE  CORRECT  CLASSIFICATION  OF  NON-MUNITIONS 

This  is  the  second  of  the  two  primary  measures  of  the  effectiveness  of  this  approach. 

3.2.1  Metric 

The  metric  for  this  objective  is  the  number  of  anomalies  from  the  master  anomaly  list  that  are 
correctly  classified  as  not-TOI. 

3.2.2  Data  Requirements 

A  list  of  anomalies  from  the  master  anomaly  list  classified  as  not  TOI  and  ground  truth  for  those 
anomalies. 

3.2.3  Success  Criteria 

The  objective  will  be  considered  to  be  met  if  75%  or  more  of  the  non-munitions  items  are 
correctly  labeled  as  not  TOI. 

3.3  STOP-DIG  THRESHOLD 

When  all  identified  anomalies  are  excavated  as  in  this  type  of  demonstration,  it  is  possible  to  tell 
the  true  classification  capabilities  of  a  classification  process  based  solely  on  a  prioritized  dig  list. 
In  a  real-world  scenario,  all  targets  may  not  be  dug  so  the  success  of  the  approach  would  depend 
on  the  ability  of  an  analyst  to  accurately  specify  a  dig/no-dig  threshold. 

3.3.1  Metric 

Percent  of  UXO  correctly  identified  as  TOI  and  numbers  of  false  alarms,  Nfa,  at  the  threshold 
are  the  metrics  for  this  objective. 

3.3.2  Data  Requirements 

A  list  of  ranked  anomalies  with  a  dig/no  dig  threshold  as  specified  by  the  demonstrator.  IDA 
personnel  will  use  their  scoring  algorithms  to  assess  the  results. 
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3.3.3  Success  Criteria 


The  objective  will  be  considered  to  be  met  if  the  threshold  between  dig  and  no-dig  places  50%  of 
the  master  list  of  anomalies  in  the  no-dig  category  while  identifying  all  UXO  30  cm  deep  or 
shallower  as  TOI  and  97%  or  more  of  UXO  57-mm  or  larger  as  TOI  when  deeper  than  30  cm. 
This  metric  assumes  approximately  ten  percent  of  all  anomalies  on  the  master  anomaly  list  are 
UXO.  If  significantly  more  than  ten  percent  are  UXO  it  may  be  necessary  to  adjust  this  metric. 

3.4  MINIMIZE  NUMBER  OF  ANOMALIES  THAT  CANNOT  BE  ANALYZED 

Anomalies  for  which  reliable  parameters  cannot  be  estimated  and  dig/no-dig  decision  rules 
cannot  be  applied  often  must  be  placed  in  the  dig  category,  thus  reducing  the  effectiveness  of  the 
classification  process. 

3.4.1  Metric 

The  number  of  anomalies  for  which  reliable  parameters  cannot  be  estimated  and  decision  rules 
applied  is  the  metric  for  this  objective. 

3.4.2  Data  Requirements 

A  list  of  all  target  parameters  along  with  a  list  of  those  anomalies  for  which  parameters  could  not 
be  reliably  estimated  and  decision  rules  applied  will  be  submitted  by  each  demonstrator. 

3.4.3  Success  Criteria 

The  objective  will  be  considered  to  be  met  if  reliable  parameters  can  be  estimated  and  decision 
rules  applied  to  90%  or  more  of  the  anomalies  on  the  master  anomaly  list. 

3.5  CORRECT  ESTIMATION  OF  TARGET  PARAMETERS 

This  objective  involves  the  accuracy  of  the  target  parameters  that  are  estimated  during  anomaly 
analysis.  Successful  classification  is  only  possible  if  the  input  features  are  internally  consistent. 
The  obvious  way  to  satisfy  this  condition  is  to  estimate  the  various  target  parameters  accurately 
and  verify  some,  or  all  of  them  with  ground  truth. 

3.5.1  Metric 

Accuracy  of  estimation  of  target  parameters  is  the  metric  for  this  objective. 

3.5.2  Data  Requirements 

A  list  of  all  target  parameters  will  be  submitted  for  each  anomaly  analyzed  as  part  of  this 
demonstration.  IDA  analysts  will  compare  these  estimated  parameters  to  those  measured  during 


the  intrusive  investigation  and  determined  via  other  means  used  by  the  ESTCP  program  office. 

3.5.3  Success  Criteria 

The  objective  will  be  considered  to  be  met  if  estimated  size  parameters  are  within  ±  20%, 
estimated  symmetry  is  correct  for  95%  of  the  cases,  estimated  X,  Y  locations  are  within  15 
cm  (la),  and  estimated  depths  are  within  10  cm  (la). 
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4.0  SITE  DESCRIPTION 


Pole  Mountain  is  located  in  southeastern  Wyoming,  approximately  7  miles  east  of  Laramie  and 
40  miles  west  of  Cheyenne,  Wyoming  in  the  Pole  Mountain  Unit  of  the  Medicine  Bow  National 
Forest  as  shown  in  Figure  5. 


Figure  5:  Site  Location  Map 
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Pole  Mountain  has  been  subdivided  into  6  munitions  response  sites  as  shown  in  Figure  6. 
The  demonstration  site  totals  50  acres  and  is  located  within  the  Bisbee  Hill  Maneuver  Area  as 
shown  in  Figure  7. 

Figure  6:  Munitions  Response  Site  Map. 


Figure  7:  Site  Location  Map. 
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4.1  SITE  HISTORY 

Pole  Mountain  was  established  in  1879  as  the  Fort  D.A.  Russell  Wood  and  Water  Reserve.  The 
land  status  alternated  between  national  forest  and  military  reservation  from  1897  to  1925.  Use  of 
Pole  Mountain  fluctuated  widely  between  1925  and  1945,  with  some  periods  of  almost  no 
military  use  and  other  periods  of  heavy  use.  Pole  Mountain,  which  has  been  known  by  a  variety 
of  names  over  the  years,  was  extensively  used  as  a  target  and  maneuver  area  before  1959  by  the 
Anny,  the  Reserve  Officers’  Training  Corps,  the  Citizens’  Military  Training  Corps,  various 
National  Guard  units,  and  the  Department  of  the  Air  Force.  Disposition  of  portions  of  Pole 
Mountain  by  the  Department  of  Defense  or  its  predecessor  agencies  included  actions  between 
1945  and  1960.  In  July  1961,  a  Public  Land  Order  terminated  ah  military  interests  in  the  Pole 
Mountain  District. 

4.2  SITE  GEOLOGY 

Pole  Mountain  and  the  surrounding  area  consist  of  grassland,  forest,  and  rock  outcrops  of  the 
Sherman  Mountains,  a  portion  of  the  Laramie  Range  located  within  the  Southern  Rocky 
Mountains  physiographic  province.  The  topography  of  Pole  Mountain  is  characterized  by  steep 
rock  outcrops  and  broad  rolling  hills,  which  are  dissected  by  drainages  that  principally  flow  in  an 
easterly  direction.  The  elevation  ranges  between  approximately  7,500  and  9,050  feet  above  mean 
sea  level. 

The  Laramie  Range  was  formed  by  folding  and  faulting  during  the  Laramide  uplift  in  which 
Precambrian  crystalline  rocks  are  exposed  in  the  core  of  the  range.  At  Pole  Mountain,  the 
exposed  core  consists  of  1.43 -billion  year-old  rocks  of  the  Sherman  Granite,  a  coarse-grained, 
metaluminous  biotite-hornblende  granite. 


12 


4.3  SITE  CONTAMINATION 


A  large  variety  of  munitions  have  been  reported  as  used  at  Pole  Mountain.  Physical  evidence  for 
the  following  items  was  discovered  during  the  Remedial  Investigation: 

•  Projectiles  containing  high  explosive  (HE)  filler  (37-mm  to  155-mm,  and  2.95-inch); 

•  Shrapnel  projectiles  (75 -mm  and  3 -inch); 

•  37-mm  projectiles  (inert  and  unfuzed) 

•  3-inch  Stokes  mortars  (practice,  fuzed); 

•  60-mm  mortars  containing  HE  filler;  and 

•  Small  arms  ammunition  (.30-caliber  and  .50-caliber) 
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5.0  TEST  DESIGN 


The  objective  of  this  program  was  to  learn  to  use  advanced  classification  processes  and  apply 
them  to  making  dig/no-dig  decisions  on  munitions  response  sites.  The  key  components  of  this 
demonstration  project  are  1)  collection  of  high  quality  geophysical  data  and  principled  selection 
of  anomalous  regions  in  those  data,  2)  analysis  of  the  selected  anomalies  using  physics-based 
models  to  extract  target  parameters  such  as  size,  shape,  and  materials  properties,  and  3)  the  use 
of  those  parameters  to  construct  a  ranked  anomaly  list.  In  the  course  of  participating  in  this 
demonstration,  the  USACE  executed  components  2  and  3  listed  above.  Target  parameters  were 
extracted  during  processing  and  passed  through  classification  routines  that  were  used  to  produce 
prioritized  anomaly  lists  ordered  from  the  item  that  the  classification  routine  determined  was 
most  likely  a  munitions  through  the  item  regarded  as  the  most  likely  to  be  nonhazardous. 

The  prioritized  anomaly  list  was  scored  by  the  IDA,  with  emphasis  on  the  number  of  items 
correctly  labeled  nonhazardous  while  correctly  labeling  all  TOIs.  The  primary  objective  of  the 
demonstration  was  to  assess  how  well  each  demonstrator  was  able  to  order  its  ranked  anomaly 
list  and  specify  the  threshold  separating  high-confidence  clutter  from  all  other  items.  The 
secondary  objective  was  to  determine  the  classification  perfonnance  that  could  be  achieved  by 
each  approach  through  a  retrospective  analysis. 

5.1  SITE  PREPARATION 

USACE  was  not  involved  in  site  preparation  for  this  project.  Site  set-up  and  logistics  were 
perfonned  by  URS,  and  details  regarding  this  aspect  of  the  project  should  be  contained  in  their 
report. 


5.2  SYSTEM  SPECIFICATION 

The  MetalMapper  sensor  and  data  acquisition  system  are  described  in  detail  in  Section  2.1.  All 
MetalMapper  data  for  this  project  were  collected  by  Sky.  Site-specific  MetalMapper 
configuration  should  be  discussed  in  detail  in  Sky’s  report. 

5.3  CALIBRATION  ACTIVITIES 

All  MetalMapper  data  for  this  project  were  collected  by  Sky.  Any  MetalMapper  calibration 
activities  perfonned  on  site  should  be  discussed  in  detail  in  Sky’s  report. 

5.4  DATA  COLLECTION  PROCEDURES 

All  MetalMapper  data  for  this  project  were  collected  by  Sky.  Specific  data  collection  activities 
perfonned  on  site  should  be  discussed  in  detail  in  Sky’s  report. 
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5.5  VALIDATION 


All  anomalies  on  the  master  list  were  excavated  by  a  team  led  by  the  URS.  Each  item 
encountered  was  identified,  photographed,  its  depth  measured,  its  location  detennined  using 
centimeter-level  GPS,  and  the  item  removed  if  possible.  These  ground  truth  data  were  used  for 
evaluation  of  the  dig  lists  submitted  by  various  analysts. 
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6.0  DATA  ANALYSIS  PLAN 


The  MetalMapper  was  used  to  collect  static  data  over  2,370  targets  identified  at  Pole  Mountain 
based  on  EM61-MK2  data.  The  processing  and  analysis  steps  that  were  used  to  generate  a  dig/no 
dig  decision  for  each  target  are  described  below. 

6.1  PREPROCESSING 

Raw  MetalMapper  data  are  collected  and  stored  as  .TEM  files.  The  MetalMapper  acquisition 
software  uses  a  convention  for  assigning  a  unique  name  to  each  data  file  without  the  need  to 
manually  enter  the  name.  The  operator  supplies  a  prefix  for  the  root  name  of  the  file  (e.g., 
“Static”).  The  acquisition  software  then  automatically  appends  a  five-character  numerical  index 
to  the  filename  prefix  to  fonn  a  unique  root  name  for  the  data  file  (e.g.,  StaticOOOOl).  The  index 
is  automatically  incremented  after  the  file  has  been  successfully  written.  Although  the  Target 
identification  (ID)  is  not  used  as  the  file  name  in  the  .TEM  file,  the  Target  ID  is  stored  in  the  file 
according  to  name  of  the  target  highlighted  on  the  MetalMapper  screen  during  collection.  Pre¬ 
processing  of  the  .TEM  files  was  performed  by  Sky  personnel  and  consisted  of  removing 
background  values  from  the  data,  converting  the  points  from  the  geographic  coordinate  system 
used  for  collection  to  the  Universal  Transverse  Mercator  Zone  13N  coordinate  system  used  for 
processing,  and  exporting  the  resulting  data  to  a  .CSV  file  that  could  be  imported  into  the  UX- 
Analyze  package  in  Geosoft’s  Oasis  Montaj  software.  The  exported  .CSV  file  name  contained 
both  the  collection  ID  and  the  Target  ID  (e.g.,  2621_Static00001_2621). 

6.2  PARAMETER  ESTIMATION 

All  MetalMapper  data  points  were  inverted  using  UX-Analyze  to  determine  modeled  parameters 
for  each  target.  These  parameters  included  the  location,  size,  and  orientation  of  the  source  object; 
the  polarizability  of  each  axis  of  the  object;  and  infonnation  regarding  the  quality  of  the  data  and 
the  relative  match  between  the  inverted  data  and  the  expected  model. 

All  target  inversion  was  perfonned  using  the  UX-Analyze  batch  processing  mode  using  both  the 
single  and  multiple  object  solvers.  The  single  object  and  multiple  object  results  were  run  through 
the  classification  scheme  to  detennine  which  method  returned  a  result  more  indicative  of  TOI. 
Although  the  multiple  object  result  may  have  approximated  the  expected  model  to  a  higher 
degree,  the  result  more  indicative  of  potential  TOI  was  used  for  target  ranking  to  be 
conservative. 

6.3  CLASSSIFICATION  AND  TRAINING 

The  polarizability  curves  developed  for  each  target  were  compared  to  a  library  of  known 
polarizability  curves  from  the  following  sources: 

•  Library  of  response  parameters  for  UXO  specific  to  Pole  Mountain,  to  include  37- 
mm,  57-mm  and  75-mm  projectiles,  stokes  mortars,  and  small  industry  standard 
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objects  (ISOs).  This  library  was  derived  from  test  pit  data  collected  during  the  course 
of  the  Metal  Mapper  data  acquisition  for  this  demonstration. 

•  Library  of  response  parameters  for  UXO  from  previous  advanced  classification 
demonstrations.  The  list  includes  a  multitude  of  UXO  items  from  37-mm  and  up  in 
size  and  small  ISOs.  This  library  was  supplied  by  SAIC. 

All  three  possible  combinations  of  the  primary  and  secondary  polarizability  curves  were  used 
when  matching  to  the  SAIC  library;  however,  only  two  of  the  three  combinations  were  used 
when  matching  to  the  Pole  Mountain  specific  library.  Example  polarizability  curves  generated 
for  this  demonstration  project  are  shown  in  Figure  8. 


Figure  8:  Typical  Polarizability  Curves 

1 


2.36  SLO  Testpit  - 1 0039m  37mm  type  A  -  flat 

Metric=0.31 71  Metric=0.2568 


Following  generation  of  the  polarizability  curves,  the  single  object  and  multiple  object  solver 
target  lists  were  ranked  separately  using  the  decision  rules  shown  in  Table  2. 
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Table  2:  Initial  Rule  Based  Decision  Logic 


Category  0:  Cannot  Analyze 

Category  1:  TOI 

Category  2:  Cannot  Decide 

Category  3:  Clutter 

Poor  fit  coefficients 

Library  metric  >  threshold 

Distance  from  flag  >  threshold 

Library  metric  <  threshold 

Unreasonable  depth 

Distance  from  array  >  threshold 

Negative  betas  ((31,|32,(33) 

Low  amplitude 

Noisy  betas  ((31,132,(33) 

The  classification  results  for  each  target  in  Categories  1  and  2  were  then  examined  by  the  data 
processor.  This  examination  was  performed  to  detennine  the  usability  of  the  decision  rule  logic. 
The  decision  rule  logic  would  be  deemed  usable  if  the  data  processor  identified  three  reasonable 
looking  polarization  curves  and  the  primary  polarization  axis  ((31)  did  not  match  the  two 
secondary  axes  ((32/(33)  for  the  Category  1  targets.  In  addition,  the  decision  rule  logic  would  also 
require  that  any  Category  1  target  did  not  visually  appear  to  belong  in  Category  2.  None  of  the 
Category  3  or  4  polarization  curves  were  examined  as  the  decision  rule  logic  was  considered  set 
for  these  two  categories.  Figure  9  shows  an  example  of  a  polarization  curve  where  the  initial 
decision  logic  placed  this  target  in  the  Category  1  list;  however,  it  is  plainly  evident  that  this  item 
does  not  have  the  symmetry  pattern  usually  associated  with  a  TOI. 


Figure  9:  Example  Fit  Results  for  Target  Incorrectly  Identified  as  TOI 
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Elevation  View 


2473.5 


2473  0 


2472.5 


Subsequent  to  the  visual  inspection  and  the  determination  that  the  decision  rules  were  incorrectly 
assigning  targets  to  Category  1 ,  training  data  was  requested  for  targets  predominantly  flagged  as 
Category  1  that  visually  did  not  appear  to  be  TOIs,  and  some  targets  flagged  as  Category  3  to 
confirm  certain  type  curves  as  non-TOI.  Table  3  lists  all  the  training  data  along  with  notes  on  the 
polarization  data,  which  formed  the  basis  for  changing  the  rule  based  decision  logic.  The  data 
processor  made  the  decision  not  to  further  refine  the  cutoff  values  of  the  rule  based  decision 
logic;  however,  to  concentrate  on  reducing  the  dig  list  of  Category  1  targets  by  using  the 
geometric  relationships  between  the  three  polarizations.  None  of  the  requested  targets  were 
identified  as  a  TOI  so  the  original  Pole  Mountain  library  was  not  modified  based  on  the  results  of 
the  training  data. 


Table  3:  List  of  Anomalies  for  Training  Data  Request 


Anomaly 

ID 

Identification 

Dig 

Type 

Category/Subcategory 

Polarization 

Notes 

Decision 
Rule  Notes 

1 

Decision 
Rule  Notes 

2 

PM-95 

Wire 

CD 

Cat  1 ,  Sub  6 

bl<b2  or  b3 

cutlow2  too 
low  at  0.90 

actual  = 
0.9043 

PM-95 

Other 

CD 

PM-95 

Other 

CD 

PM-95 

Other 

CD 

PM-95 

Other 

CD 

PM-90 

Other 

CD 

Catl,  Sub  6 

bl<b2  orb3 

cutlow2  too 
low  at  0.90 

actual  = 
0.9382 

PM-90 

Other 

CD 

PM-90 

Other 

CD 
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Anomaly 

ID 

Identification 

Dig 

Type 

Category/Subcategory 

Polarization 

Notes 

Decision 
Rule  Notes 

1 

Decision 
Rule  Notes 

2 

PM-90 

Wire 

CD 

PM-90 

Wire 

CD 

PM-871 

Frag  (light) 

MD 

Cat  3,  Sub  1 

Correct 

classification 

PM-830 

Horseshoe 

CD 

Cat  1,  Sub  4 

PM-830 

Other 

CD 

PM-830 

Other 

CD 

PM-830 

Nail 

CD 

PM-830 

Nail 

CD 

PM-77 

Frag  (light) 

MD 

Cat  1 ,  Sub  6 

bl<b2  or  b3 

cutlow2  too 
low  at  0.90 

actual  = 
0.9127 

PM-77 

Horseshoe 

CD 

PM-755 

Other 

CD 

Cat  1 ,  Sub  5 

b  1  nearly  equal 
to  b2,  b3  much 
less  and  betas 

cross  over 

cutlow3  too 
low  at  0.80 

actual  = 
0.8987 

PM-696 

Frag  (heavy) 

MD 

Cat  1 ,  Sub  3 

3  moderately 
spaced  betas 

middle  of 
cutlow3 
and  cuthi3 

actual  = 
0.8584 

PM-625 

Frag  (light) 

MD 

Cat  3,  Sub  1 

Correct 

classification 

PM-507 

Frag  (medium) 

MD 

Cat  3,  Sub  1 

Correct 

classification 

PM-507 

Frag  (light) 

MD 

Cat  3,  Sub  1 

Correct 

classification 

PM-503 

Nail 

CD 

Cat  1 ,  Sub  7 

3  widely  spaced 
betas 

cuthighl 
too  low  at 
0.97 

actual  = 
0.9798 

PM-491 

Other 

CD 

Cat  1,  Sub  7 

bl<b2  or  b3 

cuthighl 
too  low  at 
0.97 

actual  = 
0.9847 

PM-463 

Frag  (light) 

MD 

Cat  1,  Sub  7 

odd  decay  - 
visually  never 
would  pick 

cuthighl 
too  low  at 
0.97 

actual  = 
0.9835 

PM-433 

Other 

CD 

Cat  1 ,  Sub  6 

3  widely  spaced 
betas 

cutlow2  too 
low  at  0.90 

actual  = 
0.9317 

PM-433 

Nail 

CD 

PM-415 

Other 

CD 

Cat  1 ,  Sub  6 

3  widely  spaced 
betas 

cutlow2  too 
low  at  0.90 

actual  = 
0.9059 

PM-350 

Frag  (medium) 

MD 

Cat  1 ,  Sub  7 

cuthighl 
too  low  at 
0.97 

actual  = 
0.9734 

PM-320 

Other 

CD 

Cat  3,  Sub  1 

Correct 

classification 

PM-315 

Horseshoe 

CD 

Cat  1 ,  Sub  6 

bl<b2  or  b3 

cutlow2  too 
low  at  0.90 

actual  = 
0.9945 

PM-2319 

Other 

CD 

Cat  1,  Sub  6 

bl<b2  or  b3 

cutlow2  too 
low  at  0.90 

actual  = 
0.9804 

PM-225 

Frag  (light) 

MD 

Cat  3,  Sub  1 

Correct 

classification 

PM-2208 

Frag  (light) 

MD 

Cat  1 ,  Sub  5 

3  very  closely 
spaced  betas, 
bl<b2  or  b3 

cutlow3  too 
low  at  0.80 

actual  = 
0.8050 

PM-2208 

Frag  (light) 

MD 

PM-2208 

Frag  (light) 

MD 

PM-2208 

Frag  (light) 

MD 

PM-2208 

Frag  (light) 

MD 
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Anomaly 

ID 

Identification 

Dig 

Type 

Category/Subcategory 

Polarization 

Notes 

Decision 
Rule  Notes 

1 

Decision 
Rule  Notes 

2 

PM-2180 

Frag  (light) 

MD 

Cat  1,  Sub  5 

3  very  closely 
spaced  betas, 
bl<b2  or  b3 

cutlow3  too 
low  at  0.80 

actual  = 
0.8036 

PM-2180 

Frag  (light) 

MD 

PM-2180 

Frag  (light) 

MD 

PM-2180 

Frag  (light) 

MD 

PM-2180 

Frag  (light) 

MD 

PM-2101 

Frag  (light) 

MD 

Cat  1,  Sub  5 

3  very  closely 
spaced  betas, 
bl<b2  or  b3 

cutlow3  too 
low  at  0.80 

actual  = 
0.8737 

PM-2101 

Frag  (light) 

MD 

PM-2101 

Frag  (light) 

MD 

PM-2101 

Frag  (light) 

MD 

PM-2101 

Frag  (light) 

MD 

PM-2098 

Frag  (light) 

MD 

Cat  1 ,  Sub  5 

3  equally  spaced 
betas 

cutlow3  too 
low  at  0.80 

actual  = 
0.8140 

PM-2098 

Frag  (light) 

MD 

PM-2098 

Frag  (light) 

MD 

PM-2098 

Frag  (light) 

MD 

PM-2098 

Frag  (light) 

MD 

PM-2086 

Frag  (heavy) 

MD 

Cat  1 ,  Sub  5 

3  very  closely 
spaced  betas 

cutlow3  too 
low  at  0.80 

actual  = 
0.8979 

PM-2083 

Frag  (light) 

MD 

Cat  1,  Sub  6 

3  equally  spaced 
betas 

cutlow2  too 
low  at  0.90 

actual  = 
0.9461 

PM-2012 

Frag  (light) 

MD 

Cat  1 ,  Sub  6 

3  equally  spaced 
and  wide  apart 
betas 

cutlow2  too 
low  at  0.90 

actual  = 
0.9201 

PM-1996 

Frag  (medium) 

MD 

Cat  1 ,  Sub  5 

3  equally  spaced 
close  anomalies 

cutlow3  too 
low  at  0.80 

actual  = 
0.8429 

PM- 1992 

Frag  (light) 

MD 

Cat  1 ,  Sub  3 

low  of  of 
cutlow3 
and  cuthi3 

actual  = 
0.8104 

PM- 194 

Fuze/Fuze 

Components 

MD 

Cat  1 ,  Sub  5 

3  very  closely 
spaced  betas 

cutlow3  too 
low  at  0.80 

actual  = 
0.9423 

PM-1816 

Wire 

CD 

Cat  1,  Sub  6 

bl<b2  or  b3 

cutlow2  too 
low  at  0.90 

actual  = 
0.9031 

PM- 180 

Frag  (medium) 

MD 

Cat  1 ,  Sub  5 

b2  and  b3  spaced 
a  little  too  far 
apart 

cutlow3  too 
low  at  0.80 

actual  = 
0.8705 

PM- 1743 

Frag  (light) 

MD 

Cat  1 ,  Sub  4 

3  relatively  close 
equally  spaced 
betas 

low  of 
cutlow  2 
and  cuthi2 

actual  = 
0.9103 

PM- 1743 

Frag  (light) 

MD 

PM-1743 

Frag  (light) 

MD 

PM-1743 

Frag  (light) 

MD 

PM-1743 

Frag  (light) 

MD 

PM- 1624 

Horseshoe 

CD 

Cat  1,  Sub  5 

3  very  closely 
spaced  betas 

cutlow3  too 
low  at  0.80 

actual  = 
0.8811 

PM-1624 

Can  Fragments 

CD 

PM-1624 

Other 

CD 

PM-1624 

Other 

CD 

PM-1624 

Other 

CD 

PM- 1496 

Horseshoe 

CD 

Cat  1 ,  Sub  4 

bl<b2  or  b3 

low  of 
cutlow  2 
and  cuthi2 

actual  = 
0.9020 

PM-136 

Horseshoe 

CD 

Cat  1 ,  Sub  7 

bl<b2  or  b3 

cuthighl 
too  low  at 

actual  = 
0.9969 
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Anomaly 

ID 

Identification 

Dig 

Type 

Category/Subcategory 

Polarization 

Notes 

Decision 
Rule  Notes 

1 

Decision 
Rule  Notes 

2 

0.97 

PM-136 

Horseshoe 

CD 

PM-136 

Horseshoe 

CD 

PM-1344 

Other 

CD 

Cat  1,  Sub  5 

cutlow3  too 
low  at  0.80 

actual  = 
0.8658 

PM- 1340 

Other 

CD 

Cat  1 ,  Sub  5 

cutlow3  too 
low  at  0.80 

actual  = 
0.8350 

PM-1300 

Fuze/Fuze 

Components 

MD 

Cat  1 ,  Sub  5 

3  very  closely 
spaced  betas 

cutlow3  too 
low  at  0.80 

actual  = 
0.9201 

PM-1251 

Frag  (medium) 

MD 

Cat  1 ,  Sub  6 

3  widely  spaced 
betas,  bl<b2  or 
b3 

cutlow2  too 
low  at  0.90 

actual  = 
0.9416 

PM-1225 

Fuze/Fuze 

Components 

MD 

Cat  1 ,  Sub  5 

3  very  closely 
spaced  betas 

cutlow3  too 
low  at  0.80 

actual  = 
0.9106 

PM-119 

Nail 

CD 

Cat  1 ,  Sub  5 

cutlow3  too 
low  at  0.80 

actual  = 
0.8034 

PM-119 

Frag  (light) 

MD 

PM-1119 

Other 

MD 

Cat  1 ,  Sub  5 

3  very  closely 
spaced 
anomalies 

cutlow3  too 
low  at  0.80 

actual  = 
0.9065 

PM-1003 

Wire 

CD 

Cat  1,  Sub  5 

3  very  closely 
spaced  betas 

cutlow3  too 
low  at  0.80 

actual  = 
0.8422 

A  method  was  devised  for  calculating  the  average  polarizability  at  2  locations  along  each  of  the 
three  polarization  curves  for  each  target.  Early  beta  calculations  were  centered  on  time  gate  1 1 
and  late  beta  calculations  were  centered  on  time  gate  30,  each  with  a  window  width  of  8  time 
gates.  A  new  decision  logic  rule  was  established  based  on  the  ratio  of  the  ratios  of  the  three  betas 
at  both  early  and  late  times  in  the  polarizability  curves.  The  final  decision  logic  is  shown  in 
Table  4. 


Table  4:  Final  Rule  Based  Decision  Logic 


Category  0:  Cannot  Analyze 

Category  1:  TOI 

Category  2:  Cannot  Decide 

Category  3:  Clutter 

Poor  fit  coefficients 

Library  metric  >  threshold 

Distance  from  flag  >  threshold 

Library  metric  <  threshold 

Unreasonable  depth 

Beta  ratios  x  threshold 

Distance  from  array  >  threshold 

Negative  betas  ((31,|32,(33) 

Low  amplitude 

Noisy  betas  ((31,(32,(33) 

Final  classification  for  the  PMTMA  project  was  accomplished  by  combining  the  single  and 
multiple  object  lists  into  one  master  list,  and  using  the  revised  decision  rules  generated  after 
analysis  of  the  training  data.  Confidence  metrics  generated  for  each  target  during  the  comparison 
to  the  various  library  data  were  used  to  rank  the  anomalies  in  each  category  from  highest  to 
lowest  confidence  metric. 

The  final  dig  list  was  submitted  with  the  following  parameters: 

•  Training  Data:  43  items  selected 
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•  Cannot  Analyze  (Category  0):  41  items  selected 

•  Likely  TOI  (Category  1):  884  items  selected 

•  Cannot  Decide  (Category  2):  411  items  selected,  all  which  were  assigned  a  no-dig 
status 

•  Likely  Clutter  (Category  3):  987  items  selected 
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7.0  PERFORMANCE  ASSESSMENT 


7.1  CORRECT  CLASSIFICATION  OF  MUNITIONS 

The  IDA  compared  the  submitted  dig  list  to  ground  truth  data  from  Pole  Mountain.  The  results 
were  judged  according  to  performance  objectives  identified  for  this  project  in  the  Demonstration 
Plan  (U.S.  Army  Corps  of  Engineers,  2011). 

Figure  10  shows  the  receiver  operating  characteristics  (ROC)  curve  for  the  final  dig  list.  As 
indicated  in  the  figure,  all  TOI  were  correctly  identified  and  this  perfonnance  objective  was  met. 


Figure  10:  ROC  Curve  for  Final  Dig  List 


PoleMtn  USACE  LibMatch  None  MetalMapper  Custom  s2  v2  UXO 


No  Difficult  UXOs 


7.2  MAXIMIZE  CORRECT  CLASSIFICATION  OF  NON-MUNITIONS 

A  few  of  the  2,370  targets  for  which  data  were  collected  at  Pole  Mountain  ended  up  being 
multiple  picks  on  the  same  source,  so  a  total  of  2,368  digs  were  perfonned  during  the  project. 
The  small  reduction  in  targets  means  that  there  were  2,208  true  clutter  items  in  the  data  set.  The 
metric  set  prior  to  the  demonstration  was  to  correctly  identify  75%  of  the  non-TOI  anomalies. 
The  final  dig  list  correctly  identified  987  items  (44.7%)  of  the  clutter  items  as  clutter.  This  metric 
was  not  met  due  to  the  large  number  of  anomalies  classified  as  “Can’t  Analyze”  which  resulted 
in  additional  digs. 
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7.3  STOP  DIG  THRESHOLD 


For  the  final  dig  list  submitted,  all  TOI  at  the  site  were  correctly  identified  as  TOI,  and  the 
number  of  false  positives  was  reduced  by  more  than  50  percent  of  the  total  number  of  false 
positives  (59%).  This  exceeds  the  perfonnance  objectives  for  the  dig  threshold  and  passes  this 
perfonnance  criterion. 

7.4  MINIMIZE  NUMBER  OF  ANOMALIES  THAT  CANNOT  BE  ANALYZED 

The  final  dig  list  contained  41  targets  categorized  as  “Cannot  Analyze”.  This  corresponds  to 
1.7%  of  the  targets  at  the  site  and  exceeds  the  perfonnance  metric  established. 


7.5  CORRECT  ESTIMATION  OF  TARGET  PARAMETERS 

Target  parameters  were  not  submitted  for  analysis.  Positioning  may  have  been  a  problem  with 
the  data,  as  15%  of  the  anomalies  had  offsets  from  their  model  positions  greater  than  0.5m  and 
were  subsequently  classified  as  “Can’t  Analyze.” 


7.6  RETROSPECTIVE  ANALYSIS 

The  ROC  curve  portrayed  in  Figure  10  indicates  three  areas  where  improvements  to  the 
advanced  classification  analysis  could  be  made.  Above  approximately  95%  of  UXO  correctly 
classified,  the  classification  scheme  results  in  a  large  number  of  debris  (munitions  and  cultural) 
being  incorrectly  classified.  The  second  issue  is  the  last  TOI  correctly  identified  falls  very  far 
down  the  list.  Lastly,  the  number  of  “cannot  decide”  items  is  quite  substantial.  The  first  two 
issues  are  mainly  related  to  the  use  of  a  target  library  which  contains  many  more  munitions  items 
than  has  been  found  at  the  Pole  Mountain  Site  and  the  fact  that  the  classification  scheme 
iteratively  looks  for  these  anomaly  types  much  later  in  the  routine. 

In  keeping  with  the  classification  approach  which  did  not  include  feature  space  analysis,  the  data 
for  a  select  number  of  key  attributes  (axial  and  plate  symmetry,  brat,  bsum,  etc)  were  plotted  by 
classification  type  (Figure  11)  to  look  for  a  trend  which  may  help  correct  the  issues  identified 
above. 
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Figure  11:  Parameter  Trend  Analysis 


It  is  evident  that  the  bsum  parameter,  which  is  related  to  the  size  of  the  item,  had  a  cutoff  set  way 
too  low.  By  appropriately  adjusting  this  parameter,  the  last  TOI  correctly  classified  can  be  found 
much  earlier  in  the  dig  list  and  the  number  of  incorrectly  identified  TOI  are  also  reduced  as 
shown  in  Figure  12. 


Figure  12:  ROC  curve  for  conservative  readjustment  of  bsum  parameter 


By  using  the  most  aggressive  classification  scheme  by  applying  the  new  bsum  cutoff  to  all 
anomalies,  the  number  of  “cannot  decide”  items  is  also  greatly  reduced  as  shown  in  Figure  13. 
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Figure  13:  ROC  curve  for  most  aggressive  readjustment  of  bsum  parameter 


The  dig  list  for  the  ROC  curve  shown  in  Figure  13  had  the  following  statistics: 

•  Training  Data:  43  items  selected 

•  Cannot  Analyze  (Category  0):  41  items  selected 

•  Likely  TOI  (Category  1):  453  items  selected 

•  Cannot  Decide  (Category  2):  66  items  selected,  all  which  were  assigned  a  no-dig 
status 

•  Likely  Clutter  (Category  3):  1763  items  selected 
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8.0  COST  ASSESSMENT 


Costs  were  broken  down  into  three  categories:  Learning,  Analysis  and  Application,  and 
Reporting.  Learning  includes  all  activities  undertaken  to  understand  and  implement  the  new 
advanced  classification  tools,  and  includes  a  3-day  trip  to  Ft.  Ord  for  hands-on  MetalMapper 
training. 

Analysis  and  Application  includes  the  time  required  to  perform  inversions  using  UX-Analyze 
and  to  create  decision  plots  for  each  target,  examine  the  polarization  curves,  identify  unknown 
items  for  training  data  request,  and  finalize  the  classification  scheme. 


Table  5:  Cost  Assessment  Table 


Cost  Element 

Cost/Quantity 

Learning 

$3. 91/anomaly 

Analysis  and  Application 

$7. 17/anomaly 

Reporting 

$  1.69/anomaly 
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